Skip to content

Add flexible cluster and worker management and lots of other stuff #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 65 commits into from
May 10, 2024
Merged

Add flexible cluster and worker management and lots of other stuff #13

merged 65 commits into from
May 10, 2024

Conversation

@oschulz
Copy link
Owner

@oschulz oschulz commented May 10, 2024

No description provided.

oschulz added 30 commits April 27, 2024 19:16
@oschulz
Add on_free_worker (Julia >= 1v.9 only)
edb2653
@oschulz
Use collect for Cmd components
ae69c79
@oschulz
Better logging handling for test_onworkers
43ce4d3
@oschulz
Make on_free_worker throw original exception
175eb4c
@oschulz
Disable ThreadPinning OS warnings in tests
ad88613
@oschulz
Fixes for non-Linux platforms
0866c97
@oschulz
Small fix in _on_free_worker_impl_
cd91fee
@oschulz
Fix sleep tests on non-Linux OS, esp. OS-X
3eb4f2d
@oschulz
Add getlabel, isactive, hasfailed and whyfailed
f40c1c4
@oschulz
Improve start_elastic_workers
9bafd64
@oschulz
Rename util.jl to waiting.jl
61edac2
@oschulz
Add wouldwait
8498265
@oschulz
Add macro wait_while and functions wait_for_any and wait_for_all
6d480fb
@oschulz
Require ArgCheck and Sockets
3869385
@oschulz
Remove unused _addprocs_slurm
9dd2556
@oschulz
Improve wait_for_any and wait_for_all
d9e3662
@oschulz
Support Nothing in state checks
315c084
@oschulz
Remove worker scheduler, add new procinit and FlexWorkerPool
8a7f83c
@oschulz
Set atomic file operations default verbosity to false
f6b243a
@oschulz
Handle MethodError in onwoker
de678f0
@oschulz
Fix test_states
3d6653a
@oschulz
Adds inner_exception, original_exception get's a new role
4897aea
@oschulz
Adapt to change from original_exception to inner_exception
35f082f
@oschulz
Add macro return_exceptions
d2a4804
@oschulz
Add function default_cache_dir
24800c3
@oschulz
Add function modify_files
42c972a
@oschulz
Make worker_resources robust
b3d9b44
@oschulz
Don't init workers in addworker
ad4e110
@oschulz
Improve take! implementation for FlexWorkerPool
a8517c0
@oschulz
No greedy worker init, add occupancy info to FlexWorkerPool
669870a
oschulz added 24 commits May 7, 2024 09:44
@oschulz
Add function stopworkers
acda3b9
@oschulz
Multiple implementation and test fixes.
67f3192
@oschulz
Improve write_worker_start_script implementation
7a775eb
@oschulz
Change worker_start_command and write_worker_start_script
ad58cb4
@oschulz
Fix handling of worker_timeout options
bd6f357
@oschulz
Add timeout support to waiting functions
1dfc4a4
@oschulz
Use wait_for_any timeout in _on_worker_impl
deb174e
@oschulz
Adapt to changes in ElasticManager
cb21ec9
@oschulz
Use internal modified version of ElasticManager for now
8440abf 
Switch to ClusterManagers.ElasticManager once changes have been upstreamed.
@oschulz
Make write_worker_start_script return script filename
b240451
@oschulz
Add testtools
5225a16
@oschulz
Improve onworkers tests
25f942e
@oschulz
Move ThreadPinning related code to into an extension
6d8aa75
@oschulz
Fix tests and enable onworker on Julia v1.6
08e41a4
@oschulz
FIx global_procinit_level docstring
f2f62f5
@oschulz
Remove on_free_worker
640f27e
@oschulz
Fix docstring of idle_sleep
0cfb093
@oschulz
Remove keep_showing
fff214e
@oschulz
Run ensure_procinit before worker_resources
ecd569a
@oschulz
Add package functionality overview to docs
422ef0b
@oschulz
Relax sleep timing tests on OS-X a lot
1623896
@oschulz
Deactivate write_worker_start_script tests
4e48ae6 
Fails during CI testing for some reason.
@oschulz
Set JULIA_REVISE to off on elastic workers
3d0729f 
Revise has Distributed support, workers shouldn't run Revise separately.
@oschulz
Fix NonZeroExitCode ctor
b8ad012
@oschulz oschulz mentioned this pull request May 10, 2024
Closed
@codecov
Copy link

codecov bot commented May 10, 2024

Codecov Report

Attention: Patch coverage is 71.53025% with 240 lines in your changes are missing coverage. Please review.

Project coverage is 70.00%. Comparing base (eb1cd88) to head (b8ad012).

Files Patch % Lines
src/workerpool.jl 70.80% 47 Missing ⚠️
src/custom_cluster_managers.jl 55.68% 39 Missing ⚠️
src/runworkers.jl 63.44% 34 Missing ⚠️
src/display.jl 0.00% 28 Missing ⚠️
src/onworkers.jl 71.42% 28 Missing ⚠️
src/procinit.jl 82.17% 23 Missing ⚠️
src/slurm.jl 0.00% 17 Missing ⚠️
ext/ParallelProcessingToolsThreadPinningExt.jl 0.00% 12 Missing ⚠️
src/exceptions.jl 69.23% 4 Missing ⚠️
src/waiting.jl 96.33% 4 Missing ⚠️
... and 2 more
Additional details and impacted files 
@@             Coverage Diff             @@
##             main      #13       +/-   ##
===========================================
+ Coverage   48.31%   70.00%   +21.69%     
===========================================
  Files          11       19        +8     
  Lines         563     1227      +664     
===========================================
+ Hits          272      859      +587     
- Misses        291      368       +77

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@oschulz oschulz merged commit b949759 into main May 10, 2024
@oschulz oschulz deleted the clusters-and-workers branch May 10, 2024 19:18
@oschulz
Copy link
Owner Author

oschulz commented May 10, 2024

CC @JBlaschke this should be the main implementation for now - will probably do another PR changing some things before I make a release.

@oschulz
Copy link
Owner Author

oschulz commented May 10, 2024

I also want to see if I can add support for Dagger Contexts (https://juliaparallel.org/Dagger.jl/dev/processors/#Dynamic-worker-pools) , to automatically add/remove workers there via the mechanisms in ParallelProcessingTools.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@oschulz